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INTRODUCTION 

The paper by Bedford, Quigley and Walls has re- 
viewed the role of expert judgements in reliability 
assessment within the engineering design process. It 
differs from many statistically based papers that fo- 
cus mainly on statistical models, in that it addresses 
issues related to the process, the concept and prin- 
ciples. The paper itself is comprehensive in coverage 
and offers general guidance for reliability assessment 
at the system design stage. However, to some ex- 
tent, the technical side of the statistical techniques 
used in expert opinion elicitation in reliability anal- 
ysis has not been fully explored. A practitioner who 
wants to conduct research in the area still needs 
to search methodologies from other recommended 
sources in the paper to find practical solutions. This 
is perhaps the purpose of the paper. 

In my discussion, two techniques that are partic- 
ularly useful to subjective data elicitation, but are 
not discussed in the Bedford, Quigley and Walls pa- 
per, namely, empirical Bayes (EB; Robbins, 1955) 
and evidential reasoning (ER; Dempster, 1966), will 
be briefly discussed. It is hoped that this will shed 
some light on the problem of expert data elicita- 
tion in general and reliability assessment in design 
in particular. 

THE EB AND ER APPROACHES 

Suppose that a prior distribution is hypothesized 
for the parameter G, but is only specified to be in 
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a certain class of prior distributions. We can repre- 
sent this by a prior distribution p(0|<fr), where the 
hyperparameters <3? index the family of priors. We 
can then construct a "likelihood" 

(1) p(x\$) = J p{x\0)p{e\<5>) dO 

that relates the data to the hyperparameters. In the 
so-called empirical Bayes approach, <1> is estimated 
from (1) by classical methods such as unbiased es- 
timation, yielding an estimate The prior is then 
taken to be p(@\<&) and the inference about is by 
the appropriate Bayes rule for this prior. The anal- 
ysis thus has the flavor of a Bayesian analysis, but 
with an empirical prior based on the data, so it is 
termed empirical Bayes. If we view x as the sub- 
jective or expert data, then the EB approach can 
be readily used in the reliability analysis of design 
problems. 

The Dempster-Shafer theory of evidence (Demp- 
ster, 1966; Shafer, 1976) was particularly developed 
to aggregate subjective assessments in decision- 
making. To begin with, suppose there is a simple 
two-level hierarchy of attributes with a general at- 
tribute at the top and a number of basic attributes 
at the bottom. In the reliability assessment of a de- 
signed system, the basic attributes can be viewed as 
the components of the system and the general at- 
tribute can be the system reliability. Let P n ^ be a 
probability mass that represents the degree to which 
the ith basic attribute supports the hypothesis that 
the general attribute is assessed to the nth grade. 
Then P n j is calculated as 
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(2) P, 

where u>i is the weight of the ith attribute and j3 n j 
is the degree of belief that the ith basic attribute 
supports the hypothesis that the general attribute 
is assessed to the nth grade. Define a subset of the 
basic attributes and let P n ,l(i) denote the probability 
mass defined as the degree to which all the attributes 
in the subset support the hypothesis that the general 
attribute is assessed to the nth grade. Then we have 
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where Kju+y is a normalizing factor. In the origi- 
nal ER approach, the combined degree of belief /3 n is 
given by 

(4) Pn = P n ,I(Lh 

where L is the number of basic attributes. 

For simplicity, (3) is presented by assuming 
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1, where N is the number of grades. The 



case of J2iLi Pn,i < 1 w as actually established in the 
original ER algorithm (Shafer, 1976). 

From this brief introduction to the EB and ER 
approaches, we can see that they are relevant to 
reliability assessment using subjective expert data. 
In the next section, we further discuss these two 
techniques with respect to the reliability assessment 
problem at the design stage. 

EB AND ER APPROACHES IN RELIABILITY 
ASSESSMENT 

We start with the EB approach, which is mainly 
concerned with the Bayesian treatment of the data 
collected for reliability assessment. 

It is noted that the most difficult problem in sub- 
jective data acquisition is formulating the type of 
questions designed to collect useful data from the 
experts. I agree with the authors that the best ap- 
proach is to ask the experts information about their 
observed events, which is typically done in prob- 
abilistic risk assessment (PRA). Of course, at the 
design stage many events are unknown; therefore, 
we have to extract information from the experts 
about the unknown quantity. Empirical Bayes meth- 
ods may be of use in this context; see Lemon (1972), 
Yiiceer and Inoue (1980), Tillman, Kuo, Hwang and 
Grosh (1982), Savchuk (1988), Ho and Pinheiro (2002) 
Denson, Keene and Caroli (1998), Vasconcelos and 
Lippman (1997) and Wang and Jia (2006). 

The EB technique uses information from observed 
events, but the aim is to make inference on the un- 
observed quantity of interest. In standard Bayesian 
methods, priors are typically chosen so as to min- 
imize computational complexity or are set to arbi- 
trary values. While using empirical Bayes methods, 
the hyperparameters in the prior are obtained by 
maximizing the marginal distribution of some ob- 
served events conditional on the hyperparameters; 
see (1). This approach may violate the fundamen- 
tal Bayesian principle that priors should not be es- 
timated from the data, but in practice it leads to 
more sensible solutions than setting priors arbitrar- 
ily or using priors whose main justification comes 



from computational simplicity (the so-called con- 
jugate priors). Wang and Jia (2006) have demon- 
strated this point using simulated and real data. 
The empirical Bayes approach also provides a way to 
break the infinite chain of conditional probabilities 
involved in standard hierarchical Bayesian inference, 
while it still allows for different priors, depending on 
the context. The EB approach is an approximation 
to the standard Bayesian analysis, but it still retains 
some flavor of the Bayesian treatment as explained 
before. The main advantage of the EB approach over 
standard Bayesian analysis is the reduction of com- 
puting complexity and time requirements. This is 
particularly useful when handling a design problem 
with many unknown factors. The accuracy of such 
approximation depends on the case considered and 
the shapes of the prior distributions. 

Consider the following example: If we are inter- 
ested in design reliability d, which is characterized 
by an unknown parameter and a prior distribu- 
tion p(0|$), where $ are the hyperparameters of 
the prior, it follows that 



= / p(d\@)p(G\$)d@. 



(5) 



Suppose that a set of expert data x is available. 
Then, using Bayes' theorem, we have 



(6) 



p(Q\x) 



p(x\e) P (&) 

Jp(x\Q)p(e)dQ- 



Since is unknown in a standard framework of 
Bayesian inference unless there is absolute certainty 
regarding the values of x, we have to use 



(7) p(Q\x) 



/p(x|9)p(e|$)p($)d$ 
J Jp(x\e)p(G\<f>)p($)dxd$ 



instead of (6). This will, of course, involve a signifi- 
cant increase in complexity. 

The empirical Bayes perspective is to avoid this 
increase by keeping p(0|$) but choosing the param- 
eter set $ that best explains the data x from (1). 
Inferences are then based on (6) using these esti- 
mated values, namely, 



(8) 



p(0|x,$) 



p(x\e)p{e\$) 



/p(x|e)p(e|*)de 

Equation (5) then becomes 



(9) 



p(d|$)= / p(d\Q)p(@\x,<S>)de. 
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Now we discuss the potential use of ER in reli- 
ability assessment at the design stage. The ER ap- 
proach is particularly useful in reliability assessment 
when the data collected are imprecise and incom- 
plete, which is often encountered in subjective data 
elicitation. 

Data obtained from experts are typically impre- 
cise and incomplete. This is a key issue in subjective 
data elicitation. This topic is hot, particularly in 
the area of decision-making using subjective data; 
see, for example, Athanassopoulos and Podinovski 
(1997), Bana e Costa and Vincke (1995), Miettinen 
and Salminen (1999) and Weber (1987). Coolen and 
Newby (1994) also provides a good treatment of the 
same issue in reliability analysis from a Bayesian 
point of view. Imprecise information means that the 
information provided does not impose a precise com- 
bination of the values for the parameters of inter- 
est. A new designation for this type of information 
is the name nonpoint information, which implies 
that multiple scenarios and interval data are typ- 
ically the type of data collected. A good deal of re- 
search carried out in multi-criteria decision-making 
using evidential reasoning (ER) could be of use in 
this context of reliability assessment; see Yang and 
Xu (2002), Palacharla and Nelson (1994), Saffiotti 
(1994), Dempster (1966), Dempster and Kong (1986) 
and Shafer (1976). One feature of ER is its ability to 
handle qualitative data, which can also be useful in 
reliability assessment. Suppose for now that the new 
system under design is a motorcycle. To evaluate the 
reliability of this system, the general attribute can 
be the reliability of the motorcycle, and the three ba- 
sic attributes are assumed to be engine, transmission 
and brakes. The set of evaluation grades is defined 
byH = {poor, indifferent, average, good, excellent}. 
Reliability is a general technical concept and it is 
difficult to assess directly. It needs to be decom- 
posed into detailed subconcepts. Even the concept of 
brakes must be measured by stopping power, brak- 
ing stability and feel at control. Now if expert data 
become available with respect to an expert's per- 
sonal degree of belief, f3 n i, that the ith basic at- 
tribute supports the hypothesis that the general at- 
tribute is assessed to the nth grade and the weight 
u% is also available, then by utilizing (2)-(4), the reli- 
ability of the new motorcycle can be assessed. There 
is a considerable amount of literature on the selec- 
tion of appropriate weights; see Hwang and Yoon 
(1981), Saaty (1988) and Yang, Deng and Xu (2001). 



Reasoning about uncertainty emphasizes the prac- 
tical application of theoretically sound techniques 
for reasoning from evidence based on information 
that is potentially incomplete, inexact, inaccurate 
and from diverse sources. Although there is very lit- 
tle reported research on the application of ER to 
reliability assessment, the ER technique has been 
widely used in Bayesian belief networks (as men- 
tioned in the paper by Bedford, Quigley and Walls). 
In fact, the probabilistic theory behind ER is Bayesian, 
but ER also uses many techniques other than proba- 
bilities, such as possibility and fuzziness. The prob- 
ability structure employed in ER is basically non- 
parametric and if a parametric form is used, EB can 
also be used. 

CONCLUSION 

This discussion provides additional insight with 
regard to expert data elicitation in general, and ap- 
plications to reliability assessment at the design stage 
in particular. The Bedford, Quigley and Walls pa- 
per covers the subject in a broad way, following a 
top-down approach. It outlines the general proce- 
dure, structure and issues related to the concepts in 
expert elicitation for reliability system design. It is 
a useful addition to the existing literature, but, as 
stated before, a detailed description of the statisti- 
cal techniques used in expert elicitation can also be 
important. 
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